Skip to content

chore: update quickstart to use new llmjudge format - #6

Merged
Cali0707 merged 1 commit into
mcpchecker:mainfrom
Cali0707:fix-quickstart-llmjudge
Apr 1, 2026
Merged

chore: update quickstart to use new llmjudge format#6
Cali0707 merged 1 commit into
mcpchecker:mainfrom
Cali0707:fix-quickstart-llmjudge

Conversation

@Cali0707

@Cali0707 Cali0707 commented Apr 1, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Contributor

Summary by CodeRabbit

  • Refactor
    • Restructured agent configuration management to use file-based references for improved organization and maintainability
    • Updated evaluation configurations to load agent and judge settings from external files instead of environment variables
    • Applied configuration improvements across multiple example iterations demonstrating best practices

@Cali0707
chore: update quickstart to use new llmjudge format
065dbd5 
Signed-off-by: Calum Murray <cmurray@redhat.com>
@Cali0707
Cali0707 merged commit d473dca into mcpchecker:main Apr 1, 2026
1 check was pending
@coderabbitai

coderabbitai Bot commented Apr 1, 2026 •
edited
Loading

Copy link
Copy Markdown

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 842ccc63-3d08-4883-b72d-05187e58e6a4

📥 Commits

Reviewing files that changed from the base of the PR and between 2925f22 and 065dbd5.

📒 Files selected for processing (4)
  • 03-evolution-case-study/iteration-1-bad/evals/agent.yaml
  • 03-evolution-case-study/iteration-1-bad/evals/eval.yaml
  • 03-evolution-case-study/iteration-2-good/evals/agent.yaml
  • 03-evolution-case-study/iteration-2-good/evals/eval.yaml
📝 Walkthrough

Walkthrough

The changes introduce agent configuration files and update evaluation manifests across two iteration directories, transitioning from environment-variable-based agent and judge configuration to externalized file-based YAML references.

Changes

Cohort / File(s) Summary
Iteration 1 (Bad) - Agent Setup
03-evolution-case-study/iteration-1-bad/evals/agent.yaml		 New Agent resource manifest defining claude-code-acp agent with acp.cmd: "claude-agent-acp".
Iteration 1 (Bad) - Evaluation Config
03-evolution-case-study/iteration-1-bad/evals/eval.yaml		 Modified eval configuration to reference agent.yaml file for agent setup and llmJudge config, replacing environment-variable-based settings.
Iteration 2 (Good) - Agent Setup
03-evolution-case-study/iteration-2-good/evals/agent.yaml		 New Agent resource manifest defining claude-code-acp agent with acp.cmd: "claude-agent-acp".
Iteration 2 (Good) - Evaluation Config
03-evolution-case-study/iteration-2-good/evals/eval.yaml		 Modified eval configuration to reference agent.yaml file for agent setup and llmJudge config, replacing environment-variable-based settings.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 From env vars scattered wide and far,
Configuration hops to files—a star!
Two iterations show the dance of change,
From chaotic env to manifest range.
Better organized, cleaner, more sane,
The rabbit approves this refactored lane! ✨

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Cali0707